Software engineers work on software code functionality and try to ensure that any changes they make to software application code improves the software code and does not introduce problems into the software application. Therefore, software engineers extensively test and verify software code changes before allowing the changes to be executed live in a production version of the software application. The ability to verify and make new changes available to end users quickly is often considered important since software applications may need to be modified frequently in order to be most effective. For example, an anomaly detection application may need to update regularly to incorporate new anomaly definitions in order to successfully detect new versions of anomalies.
Since new anomaly attacks surface frequently and the traffic patterns of the existing attacks change on a daily basis, the ability to rapidly deploy and launch new anomaly classifications is critical to maximize the effectiveness and accuracy of the anomaly detection system. Similarly, many large-scale streaming applications have classification and labeling operators in their pipelines that need to be updated repeatedly. To create accurate classification and labeling operators, applications usually include aggregators which collect statistics about specific events and data over a long period of time, markers which label events, and mergers which merge marked events.
Customary development/testing processes include creating a developer testing framework to run a streaming application pipeline in an isolated environment, separate from the production environment. In this isolated environment, software code changes can be tested and verified. Any significant change to software code requires marking, merging, and aggregating data in the isolated environment. Testing changes in an isolated environment may take a long time because an aggregation process may need various windows of time of log files which contain recordings of the events that occur as an application is executing in order to accurately determine how the system, with new changes, is working. Once the changes have been verified, a software engineer may determine that the changes that will improve the software code's execution can be pushed to the testing or production environment. In addition to this process taking a long time, the process also requires a lot of resources since the production environmental setup needs to be duplicated in the development environment. Developmental testing also requires domain specific knowledge to operate the whole application pipeline in a development environment and parse the output data.
Another customary technique for testing is to create a test pipeline which may be a replica of the production pipeline. This testing pipeline may be used for stability and non-functional testing prior to pushing changes to production. As a result, the test pipeline is typically limited to pre-verified changes.
As recognized by the inventor, there needs to be a testing framework for assessing changes to a software application, specifically configuration parameter modifications that affect the application, which allows the changes to be quickly analyzed and verified so that the changes can easily be promoted to a production environment.